Liveness and spoof detection for ultrasonic fingerprint sensors

ABSTRACT

Embodiments of apparatuses and methods for detecting a spoof finger are disclosed. In one embodiment, an ultrasonic fingerprint sensor comprises an ultrasonic transmitter configured to transmit an ultrasonic wave to a finger, an ultrasonic sensor array configured to receive a reflected ultrasonic wave from the finger, and a controller configured to determine a reflected acoustic energy of the finger based on a difference between average amplitudes of the reflected ultrasonic wave from ridges and valleys of the finger; and determine whether the finger is a spoof based at least in part on the reflected acoustic energy of the finger.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. provisional application No.62/295,490, “Liveness Detection” filed Feb. 15, 2016. The aforementionedUnited States application is hereby incorporated by reference in itsentirety.

FIELD

The present disclosure relates to the field of image processing. Inparticular, the present disclosure relates to apparatuses and methodsfor detecting a spoof finger.

BACKGROUND

Fingerprint sensing and matching is a commonly used technique forpersonal identification or verification. For example, one approach tofingerprint identification involves scanning a sample fingerprint or animage with a biometric reader/sensor and storing the image and/or uniquecharacteristics of the fingerprint image. The characteristics of asample fingerprint may then be compared to information for referencefingerprints already in a database to determine proper identification ofa person, such as for verification purposes.

Biometric sensors, particularly fingerprint biometric sensors, aregenerally prone to being defeated by various forms of spoof samples. Inthe case of fingerprint readers, a variety of techniques are known forpresenting readers with a fingerprint pattern of an authorized user thatis embedded in some kind of inanimate material such as paper, gelatin,epoxy, latex, or the like. Thus, even if a fingerprint reader can beconsidered to reliably determine the presence or absence of a matchingfingerprint pattern, it is also critical to the overall system securityto ensure that the matching pattern is being acquired from a genuine,living finger, which may be difficult to ascertain with many commonsensors.

There are various conventional solutions available in distinguish spoofsamples from a live finger. One approach is described in U.S. Pat. No.7,433,729 B2, where a technique of spoof detection is disclosed usinginfrared sensors. Another approach is described in PCT1US2015/017557,where a technique of spoof detection is disclosed using opticalwavelengths to capture characteristics of skin samples being analyzed.As ultrasonic sensors have become increasingly popular in mobiledevices, it is desirable to have apparatuses and methods for livenessand spoof detection with ultrasonic fingerprint sensors.

SUMMARY

Embodiments of apparatuses and methods for detecting a spoof finger aredisclosed. In one embodiment, a method of detecting a spoof finger by anultrasonic fingerprint sensor comprises transmitting an ultrasonic waveto a finger, receiving a reflected ultrasonic wave from the finger,determining a reflected acoustic energy of the finger based on adifference between average amplitudes of the reflected ultrasonic wavefrom ridges and valleys of the finger, and determining whether thefinger is a spoof based at least in part on the reflected acousticenergy of the finger.

According to aspects of the present disclosure, the method ofdetermining the reflected acoustic energy of the finger comprisesestimating a background energy received by the ultrasonic sensor arraywithout the finger, and removing the background energy from thereflected acoustic energy of the finger. The method of determining thereflected acoustic energy of the finger further comprises detectingdiffractions of the reflected ultrasonic wave, and calculating an imagewith reduced effects of the diffractions of the reflected ultrasonicwave. The method of determining the reflected acoustic energy of thefinger further comprises detecting a non-uniform spatial response in thereflected ultrasonic wave, and adjusting the reflected acoustic energyof the finger to equalize effects of the non-uniform spatial response inthe reflected ultrasonic wave. The method of determining the reflectedacoustic energy of the finger further comprises identifying regionsrepresenting ridges and valleys of the finger, and determining thereflected acoustic energy of the finger based on the difference betweenaverage amplitudes of the reflected ultrasonic wave from the regionsrepresenting the ridges and the valleys of the finger. The method ofdetermining the reflected acoustic energy of the finger furthercomprises recording a number of early cycles of the ultrasonic wavereceived at the ultrasonic sensor array, and adjusting the reflectedacoustic energy of the finger based on the number of early cycles of theultrasonic wave received at the ultrasonic sensor array.

According to aspects of the present disclosure, the method ofdetermining whether the finger is a spoof comprises comparing thereflected acoustic energy of the finger to a threshold range, anddetermining whether the finger is a spoof based at least in part onwhether the reflected acoustic energy of the finger falls within thethreshold range. The method of determining whether the finger is a spooffurther comprises determining variations of the reflected acousticenergy of the finger over time, and determining whether the finger is aspoof based at least in part on the variations of the reflected acousticenergy of the finger over time. The method of determining whether thefinger is a spoof further comprises comparing the variations of thereflected acoustic energy of the finger over time to variations of thereflected acoustic energy of an authorized user's finger collectedduring enrollment, and determining whether the finger is a spoof basedat least in part on a result of the comparison. The method ofdetermining whether the finger is a spoof further comprises detecting achange in temperature of the finger, and determining whether the fingeris a spoof based at least in part on the change in temperature of thefinger.

In another embodiment, an ultrasonic fingerprint sensor comprises anultrasonic transmitter configured to transmit an ultrasonic wave to afinger, an ultrasonic sensor array configured to receive a reflectedultrasonic wave from the finger, and a controller configured todetermine a reflected acoustic energy of the finger based on adifference between average amplitudes of the reflected ultrasonic wavefrom ridges and valleys of the finger; and determine whether the fingeris a spoof based at least in part on the reflected acoustic energy ofthe finger.

BRIEF DESCRIPTION OF THE DRAWINGS

The aforementioned features and advantages of the disclosure, as well asadditional features and advantages thereof, will be more clearlyunderstandable after reading detailed descriptions of embodiments of thedisclosure in conjunction with the non-limiting and non-exhaustiveaspects of following drawings. Like numbers are used throughout thefigures.

FIG. 1A illustrates an exemplary flow diagram of liveness and spoofdetection using an ultrasonic fingerprint sensor according to aspects ofthe present disclosure.

FIG. 1B illustrates an exemplary implementation of determining areflected acoustic energy of the finger according to aspects of thepresent disclosure.

FIG. 2A illustrates an exemplary implementation of performing backgroundestimation and removing background noise according to aspects of thepresent disclosure.

FIG. 2B illustrates another exemplary implementation of performingbackground estimation according to aspects of the present disclosure.

FIGS. 3A-3D illustrate a method of correcting effects of diffraction ina platen layer of an ultrasonic sensor according to aspects of thepresent disclosure.

FIG. 4A illustrates an exemplary implementation of equalizing effects ofnon-uniform spatial response in a reflected ultrasonic wave according toaspects of the present disclosure; FIG. 4B illustrates exemplary imagesand waveform of the reflected ultrasonic wave of FIG. 4A.

FIGS. 5A-5B illustrate an exemplary implementation of determining thereflected acoustic energy of a finger according to aspects of thepresent disclosure.

FIGS. 5C-5D illustrates an exemplary implementation for adjusting thereflected acoustic energy according to aspects of the presentdisclosure.

FIG. 6A illustrates an exemplary implementation of determining whether afinger is a spoof according to aspects of the present disclosure.

FIG. 6B illustrates another exemplary implementation of determiningwhether a finger is a spoof according to aspects of the presentdisclosure.

FIG. 6C illustrates yet another exemplary implementation of determiningwhether a finger is a spoof according to aspects of the presentdisclosure.

FIG. 7 illustrates an exemplary block diagram of a device that may beconfigured to detect liveness of a finger according to aspects of thepresent disclosure.

FIGS. 8A-8C illustrate an example of an ultrasonic sensor according toaspects of the present disclosure.

FIG. 9A illustrates an example of a four-by-four array of sensor pixelsfor an ultrasonic sensor array. FIG. 9B illustrates an example of ahigh-level block diagram of an ultrasonic sensor system.

DESCRIPTION OF EMBODIMENTS

Embodiments of apparatuses and methods for detecting a spoof finger aredisclosed. The following descriptions are presented to enable any personskilled in the art to make and use the disclosure. Descriptions ofspecific embodiments and applications are provided only as examples.Various modifications and combinations of the examples described hereinwill be readily apparent to those skilled in the art, and the generalprinciples defined herein may be applied to other examples andapplications without departing from the scope of the disclosure. Thus,the present disclosure is not intended to be limited to the examplesdescribed and shown, but is to be accorded the scope consistent with theprinciples and features disclosed herein. The word “exemplary” or“example” is used herein to mean “serving as an example, instance, orillustration.” Any aspect or embodiment described herein as “exemplary”or as an “example” in not necessarily to be construed as preferred oradvantageous over other aspects or embodiments.

According to aspects of the present disclosure, an ultrasonicfingerprint sensor may be configured to detect a reflected acousticenergy of a finger. The ultrasonic fingerprint sensor may compute thereflected acoustic energy of the finger and determines whether thereflected acoustic energy resembles a live finger. The reflectedacoustic energy may be proportional to the amplitude of the reflectedultrasonic wave from the interface of the sensor to the material placedon it. The computed reflected acoustic energy may be either higher orlower than the reflected acoustic energy of a live finger, and may beclassified as a spoof if the reflected acoustic energy is outside athreshold range of a live finger.

According to aspects of the present disclosure, the ultrasonicfingerprint sensor may be configured to perform reflected acousticenergy criterion generation. In some embodiments, when thick platens areused, the imaging of the acoustic load may have band-selectiveproperties. In such situations, not all spatial frequencies can beimaged with a single delay of the image capturing. To account for allthe spatial frequencies, images with several delays may be combined. Insome embodiments, the output image pixels may be classified to ridgesand valleys. The classification may be based on pre-determinedthresholds that classify high value pixels and low value pixels. Imagesof valleys may be classified with values substantially the same asimages of air. The average amplitude of valleys may be subtracted fromthe average amplitude of ridges to generate amplitude metric, which maybe used to determine the reflected acoustic energy.

According to aspects of the present disclosure, fake fingers (alsoreferred to as spoofs for conciseness) and real fingers (also referredto as live fingers or fingers) have valleys that may have similarreflection factor as air. Spoofs and fingers can have differentreflected acoustic energy. Therefore, estimation of the reflectedacoustic energy can be used to differentiate between spoofs and fingers.

According to aspects of the present disclosure, the method ofdifferentiating spoofs from real fingers may comprise fingerprint imageprocessing, liveness detection, and/or calibration on target device. Theimage processing may include, but not limited to, image enhancement,calculating part of the calibration parameters, and/or featureextraction. The liveness detection can be a real time process, which mayinclude, but not limited to, decision based on threshold per device(sensor) using the features extracted in the image processing step,and/or decision based on data collected during enrollment. Thecalibration on target device may be based on device power (amplitude),features of the diagonals-target, and/or feature from enrollment.

FIG. 1A illustrates an exemplary flow diagram of liveness and spoofdetection using an ultrasonic fingerprint sensor according to aspects ofthe present disclosure. As shown in FIG. 1A, in block 102, an ultrasonictransmitter of the ultrasonic fingerprint sensor may be configured totransmit an ultrasonic wave to a finger touching a platen of theultrasonic fingerprint sensor. In block 104, an ultrasonic sensor arrayof the ultrasonic fingerprint sensor may be configured to receive areflected ultrasonic wave from the finger. In block 106, a controller ofthe ultrasonic fingerprint sensor may be configured to determine areflected acoustic energy of the finger based on a difference betweenaverage amplitudes of the reflected ultrasonic wave from ridges andvalleys of the finger. In block 108, the controller may be furtherconfigured to determine whether the finger is a spoof based at least inpart on the reflected acoustic energy of the finger.

FIG. 1B illustrates an exemplary implementation of determining areflected acoustic energy of the finger according to aspects of thepresent disclosure. The exemplary implementation shown in FIG. 1B may becontrolled by the controller of the ultrasonic fingerprint sensor asdescribed in FIG. 1A. In block 112, the controller may be configured toperform background estimation and remove background noise of theultrasonic fingerprint sensor. In block 114, the controller may beconfigured to reduce effects of diffraction of the reflected ultrasonicwave caused by the platen of the ultrasonic fingerprint sensor. In block116, the controller may be configured to equalize effects of non-uniformspatial response due to the different spatial frequencies between ridgesand valleys of different fingerprints. In block 118, the controller maybe configured to calculate a reflected acoustic energy of the fingerbased at least in part on reflected ultrasonic wave, which includes thespatial frequencies between ridges and valleys of the finger. In block120, the controller may be configured to apply a signal correct to thereflected acoustic energy based on early cycles of the transmittedultrasonic wave received directly at the ultrasonic sensor array,without substantially any effects of reflections and diffractions. Insome embodiments, it may be optional to implement each of the blocks112, 114, 116, or 120. In other words, some embodiments may notimplement any of the blocks 112, 114, 116, or 120, while otherembodiments may implement one or more of the blocks 112, 114, 116, 120,or some combinations thereof.

FIG. 2A illustrates an exemplary implementation of performing backgroundestimation and removing background noise according to aspects of thepresent disclosure. In the example of FIG. 2A, in block 202, thecontroller of the ultrasonic fingerprint sensor may be configured toestimate a background energy received by the ultrasonic sensor arraywithout the finger being present on the platen. In block 204, thecontroller may be configured to remove the estimated background energyfrom the reflected acoustic energy of the finger.

FIG. 2B illustrates another exemplary implementation of performingbackground estimation according to aspects of the present disclosure. Inblock 212, the controller of the ultrasonic fingerprint sensor may beconfigured to determine an acquisition time delay (also referred to asrange gate delay) and an ultrasonic transmitter frequency in accordancewith a variation of a current temperature from a reference temperaturefrom which an initial background estimation and an initial ultrasonictransmitter frequency are determined. In block 214, the ultrasonicsensor array of the ultrasonic fingerprint sensor may be configured toacquire background image information based on the acquisition time delayand the ultrasonic transmitter frequency. In block 216, the controllerof the ultrasonic fingerprint sensor may be configured to compute thebackground estimate using the background image information.

According to aspects of the present disclosure, background estimationmay be determined as follows:

Im_(fg) = Im_(fg_(—)on) − Im_(fg_(—)off)Im_(bg) = Im_(bg_(—)on) − Im_(bg_(—)off)

Where Im_(fg) _(_) _(on) is the image captured with a finger or spoof onthe platen of the ultrasonic fingerprint sensor with the ultrasonicsignal being activated; Im_(fg) _(_) _(off) is the image captured with afinger or spoof on the platen of the ultrasonic fingerprint sensor withthe ultrasonic signal being disabled. Im_(bg) _(_) _(on) is the imagecaptured without any object on the platen of the ultrasonic fingerprintsensor with the ultrasonic signal being activated, Im_(bg) _(_) _(off)is the image captured without any object on the platen of the ultrasonicfingerprint sensor with the ultrasonic signal being disabled.

In one embodiment, an estimate of the background image may be obtainedby subtracting the background image Im_(bg) from the foreground imageIm_(fg). In another embodiment, an estimation of the background imagemay be obtained by projecting the foreground image on an orthogonalbasis that spreads the space of recorded background images. Thisestimation is then subtracted from Im_(fg) to produce the fingerprintimage.

According to aspects of the present disclosure, received reflectedultrasonic wave can be further processed by a diffraction mitigationprocessing engine. This diffraction mitigation processing engine (alsoreferred to as point spread function) de-blurs and corrects thediffraction artifacts.

FIG. 3A illustrates a method of correcting effects of diffraction in aplaten layer of an ultrasonic sensor according to aspects of the presentdisclosure. As shown in FIG. 3A, in block 302, the method captures aplurality of images of a fingerprint having a plurality of phases in atime sequence. In block 304, the method sums the plurality of imagesmultiplied by a complex phase exponential to form an integrated compleximage. With this approach, the time dependent part of the impulseresponse cos(2πft) can be removed.

In one embodiment, to remove the time dependent part, I1(x,y) may becomputed as follows:I ₁(x,y)=∫I(x,y,t)e ^(−2πift) dtwhere the integration is performed over values of t in the range ofI(x,y,t), to obtain:

$\begin{matrix}{{I_{1}\left( {x,y} \right)} = {{\left( {\int{{\cos\left( {{2\pi\;{ft}} - {2\pi\;{fr}\text{/}c} + \varphi} \right)}{\mathbb{e}}^{{- 2}\pi\;{ift}}{\mathbb{d}t}}} \right)*{f\left( {x,y} \right)}} + {n_{1}\left( {x,y} \right)}}} \\{= {{{{const} \cdot \left( {{\exp\left( {{{- 2}\pi\;{ifr}\text{/}c} + {i\;\varphi}} \right)} + {O\left( f^{- 1} \right)}} \right)}*{f\left( {x,y} \right)}} + {n_{1}\left( {x,y} \right)}}}\end{matrix}$Note that if the integration is over all time, then the value of (f⁻¹)term may be small and negligible. In addition, the signal to noise ratio(SNR) of I₁(x,y) can be higher than that of I(x,y,t) for a single t by afactor of Δt, the range of integration.

Note that I₁(x,y) is a complex image. The real part of I₁(x,y) and theimaginary parts of I₁(x,y) can complement each other: where the realpart of I₁(x,y) has clouds, the imaginary part of I₁(x,y) does not, orvice versa. These characteristics are shown in FIG. 3B. FIG. 3Billustrates an exemplary input complex image 312, a real part 314 of theinput complex image, and an imaginary part 316 of the input compleximage, according to aspects of the present disclosure.

Note that in FIG. 3B, the real part of I1(x,y) has the same phase as theinput image, while the imaginary part of I1(x,y) may be 90° out of phaseand complements the real part of I1(x,y). Also note that the signal tonoise ratio (SNR) of the images has improved. Since the real andimaginary parts complement each other, the complex image has no clouds,even though the real and imaginary parts do have clouds.

The reflected ultrasonic wave comprises a plurality of images of thefinger having a plurality of phases in the time sequence. The pluralityof images of a fingerprint may be combined. In some implementations, aprocessor or processing logic, as a part of the controller of theultrasonic fingerprint sensor, may be configured to integrate theplurality of images multiplied by the complex phase exponential overtime to generate the integrated complex image. Note that the integratedcomplex image includes a real image part and an imaginary image part,and where the real image part and the imaginary image part may haveapproximately 90 degrees offset in phase.

In some embodiments, an integrated complex image may be aligned to apre-selected phase. For example, a processor or processing logic may beconfigured to convolve the integrated complex image with a complexkernel, where the complex kernel is an inverse of a spatial part of animpulse response. In some embodiments, the processor or processing logicmay be optionally/additionally configured to separate spatial componentsfrom phase components of the integrated complex image using otherapproaches. In addition, the processor or processing logic may beconfigured to remove edge effects by performing the convolution usingDiscrete Cosine Transform (DCT). In some embodiments, the processor orprocessing logic may be optionally/additionally configured to removeedge effect by using a duplicate of the integrated complex image alongan edge of the integrated complex image.

In some embodiments, a maximum energy image may be determined torepresent a fingerprint according to aspects of the present disclosure.For example, the processor or processing logic may be configured tocompute a derivative of energy of the aligned complex image with respectto phase. Moreover, the processor or processing logic may be configuredto compute the maximum energy phase by setting the derivative of theenergy of the aligned complex image with respect to phase to zero.Furthermore, the processor or processing logic may be configured toassign a real component of the aligned complex image at the maximumenergy phase to be the maximum energy image.

Referring to FIG. 3C, in the frequency domain, cos(−2πfr/c+φ) andsin(−2πfr/c+φ) may be represented as 322 and 324, respectively. In thisexample, numeral 322 represents a real part of an estimated frequencyresponse of a simulated diffraction; numeral 324 represents an imaginarypart of estimated frequency response of a simulated diffractionaccording to aspects of the present disclosure.

Numeral 326 represents a sum of 322 and 324 according to aspects of thepresent disclosure. As shown in FIG. 3C, the black rings in the spectrumof the real and imaginary parts correspond to the position of theclouds, while the spectrum of the complex exponential is much smoother.In addition, the position of the black rings changes when φ is changed,while the spectrum of the exponential may be the same, only the phasechanges. Another advantage of the complex exponent is that it allows thespatial term to be separated from the overall phase term.

Referring back to FIG. 3A, in block 306, the method aligns theintegrated complex image to a pre-selected phase to form an alignedcomplex image. Note that, in some embodiments, the pre-selected phasemay be a matrix of phases (an image). With this approach, the spatialpart of the impulse response −2πfr/c can be removed. In the model shownabove, the output from the first step, I1(x,y), may be approximatelyequal to (up to a constant and some additive noise):I ₁(x,y)=e ^(iφ) h ₁ *f(x,y)h ₁=exp(−2πifr/c)Since h1 is known, I1(x,y) can be convolved with its inverse:h1⁻¹=

⁻¹(1/

h1)However, since the spectrum of h1 can be close to zero at some points,the following operation may be performed:h2=

⁻¹((

h1)*/|

h1|)I2(x,y)=h2*/1=e ^(iφ)

⁻¹(|

h1|)*f(x,y)The middle term is a band-pass filter without phase distortions.

In order to avoid edge effects during the convolution with h₂, in oneexemplary approach, the convolution may be performed with an image twicethe size of the original image by mirror reflecting along each edge andtaking the middle part at the end. FIG. 3D illustrates an exemplary realpart 502 of an aligned complex image and an imaginary part 504 of thealigned complex image according to aspects of the present disclosure.

Referring back to FIG. 3A, in block 308, the method determines a maximumenergy phase using the aligned complex image. In block 310, the methodcomputes a maximum energy image to represent the fingerprint based atleast in part on the aligned complex image at the maximum energy phase.With this approach, the overall phase φ can be removed. In the examplesshown in FIG. 3D, the imaginary part 334 of the aligned complex imagehas better image quality than the real part 332 of the aligned compleximage, and note that both the imaginary part 332 and the real part 334of the aligned complex image do not have phase inversions. The reason isthat φ≈±π/2 in this case. To find the optimal phase in the general case,the phase that gives maximum energy to the real part can be computed as:φ0=arg φmax{∫[Re(e ^(iφ) I2(x,y))]² dxdy} and then setI3(x,y)=Re(e ^(iφ0) I2(x,y))

In this optimization method, the expression can be differentiated withrespect to φ,Se^(2iφ)+S*e^(−2iφ)withS=∫I2(x,y)² dxdyNote that S is a complex number, not the total energy. Equating to 0gives:e ^(iφ0)=√(±|S|/S)

The positive sign is chosen because the negative sign gives a minimum.In one embodiment, I3 may be computed as:I3=(I2/√S)

Note that S has two square roots differing only by a sign. To fix theoverall sign, the correlation coefficient of I3 with one of the inputimages can be computed. In some implementations, if I3 turns out to benegative, then it may be set as: I3=−I3. An exemplary maximum energyimage is shown with numeral 336.

According to aspects of the present disclosure, parameters for the speedof sound c and rmax may be chosen. Other parameters may be set to theirknown values. In some embodiments, both c and rmax may be obtained bychoosing the parameters that produces the highest quality output images.In some implementations, for a glass layer, the values of c and rmax maybe chosen as: c=3700 m/s, rmax=1.33. For a plastic layer, the values ofc and rmax may be chosen as: c=2000 m/s, rmax=1.33. Note that the methoddescribed herein can work even if the values of the parameters may notbe exact. For example, if c is changed by ˜±20%, reasonably good imagesmay still be obtained.

In some implementations, the diffraction mitigation processing enginecan be modified by: 1) remove phase optimization; 2) remove the matchstatistic feature; and/or 3) optionally remove the histogramequalization feature, where the diffraction mitigation processing may beapplied as follows:I_(clean) =psf_combine(Im)

According to aspects of the present disclosure, to estimate theamplitude of the live/spoof fingerprint, several features from theenhanced image I_(clean) may be estimated based on their sorted data asfollows.

a) Max value: extract the mean over the trim values of valleys ofI_(clean).Max value=mean(trim(valleys)).

b) Min value: extract the mean over the trim values of ridges ofI_(clean)Min value=mean(trim(ridges)).

c) The amplitude:Amp=(Max value−Min value).

Trim values may be determined according to two thresholds L1 & L2 thatset the range of the trim data.

If x(1:N) is the sorted column vector of I_(clean) then:Max_Value=mean(x(L1:L2))

Let y=f lipud(x)Min_Value=mean(y(L1:L2))

The reflected acoustic energy may be estimated as:Amp_Value=Max_Value−Min_ValueNote that the amplitude may be independent of image inversions.

FIG. 4A illustrates an exemplary implementation of equalizing effects ofnon-uniform spatial response in a reflected ultrasonic wave according toaspects of the present disclosure. In the example of FIG. 4A, in block402, the controller of the ultrasonic fingerprint sensor may beconfigured to detect a non-uniform spatial response in the reflectedultrasonic wave. In block 404, the controller may be configured toadjust the reflected acoustic energy of the finger to equalize effectsof the non-uniform spatial response in the reflected ultrasonic wave.

FIG. 4B illustrates exemplary images and waveform of the reflectedultrasonic wave of FIG. 4A. As shown in FIG. 4B, numeral 414 representsan exemplary image prior to equalization adjustment, and numeral 412represents a corresponding waveform of the exemplary image of 414.Numeral 416 represents an exemplary image after the equalizationadjustment.

The ultrasonic fingerprint sensor may have non-uniform lines permillimeter (LPMM) range response. This may cause objects of the samematerials and different LPMMs to show different reflected acousticenergy scores. To address this situation, in some implementations, anequalization mask may be generated during factory calibration. An objectwith a wide spectral range may be used for mask generation, and thereflected ultrasonic image may be scaled by the mask, to provide thesame energy weight to all frequencies received.

FIG. 5A illustrates an exemplary implementation of determining thereflected acoustic energy of a finger according to aspects of thepresent disclosure. As shown in FIG. 5A, in block 502, the controller ofthe ultrasonic fingerprint sensor may be configured to identify regionsrepresenting ridges and valleys of the finger. In block 504, thecontroller may be configured to determine the reflected acoustic energyof the finger based on the difference between average amplitudes of thereflected ultrasonic wave from the regions representing the ridges andthe valleys of the finger.

In some embodiments, the controller of the ultrasonic fingerprint sensormay be configured to collect different images of the finger withcorresponding different time delays (also referred to as range gatedelays) and create a combined image of the finger using the differentimages of the finger. The controller may be further configured toidentify regions representing ridges and valleys of the finger using thecombined image of the finger, determining spatial frequencies betweenridges and valleys of the finger, and determine the reflected acousticenergy of the finger based at least in part on the spatial frequenciesbetween ridges and valleys of the finger. FIG. 5B illustrates a maskingplot for valleys and ridges according to aspects of the presentdisclosure.

FIG. 5C illustrates an exemplary implementation for adjusting thereflected acoustic energy according to aspects of the presentdisclosure. In the example shown in FIG. 5C, in block 512, an ultrasonicsensor array of the ultrasonic fingerprint sensor may be configured torecord a number of early cycles of the ultrasonic wave transmitted bythe ultrasonic transmitter and received at the ultrasonic sensor array.According to aspects of the present disclosure, the early cycles of theultrasonic wave represent direct path ultrasonic waves transmitteddirectly from the transmitter to the ultrasonic sensor array, withoutthe interference of reflections and refractions. In block 514, thecontroller of the ultrasonic fingerprint sensor may be configured toadjust the reflected acoustic energy of the finger based on the numberof early cycles of the ultrasonic wave received at the ultrasonic sensorarray. FIG. 5D illustrates an example of the number of early cycles ofthe ultrasonic wave, which are highlighted by the oval.

FIG. 6A illustrates an exemplary implementation of determining whether afinger is a spoof according to aspects of the present disclosure. Inblock 602, a controller of the ultrasonic fingerprint sensor may beconfigured to compare a reflected acoustic energy of the finger to athreshold range. In block 604, the controller may be configured todetermine whether the finger is a spoof based at least in part onwhether the reflected acoustic energy of the finger falls within thethreshold range.

In some embodiments, liveness decision may be performed in real time.Decisions may be made based on thresholds per sensor device using thefeatures extracted as described above, or use calibration parametersobtained during enrollment process. Note that the reflected acousticenergy of a live finger in steady state may be higher than the reflectedacoustic energy of a spoof. In addition, the reflected acoustic energyof a live finger may reach its steady state levels sooner, if its timederivative is high. On the other hand, this may not be the case forspoofs that have low reflected acoustic energy derivative.

In some embodiments, reflected acoustic energy can be calculated as afunction of time. At each time point the reflected acoustic energy canbe calculated over the cleaned image (with removal of background imageand correction of diffraction effects) and compared against twothresholds (between a low threshold and a high threshold).

If ( ImpAmp > HIGH_THR ) Decision = Finger; reliability = HIGH (90%);Close and Return from liveness else if ( ImpAmp < LOW_THR ) Decision =Spoof; reliability = HIGH (90%); Close and Return from liveness elseContinue

If the current reflected acoustic energy is between the low and highthreshold range, the method continues to the next time point.

Upon ImpAmp for all time points have been calculated, the reflectedacoustic energy derivative can be calculated as follows:

  ${slope} = {\frac{\partial({ImpAmp})}{\partial t}.}$ And apply thesimple logic:${If}\mspace{14mu}\left( {\left( {{{avg}\left( \frac{\partial({ImpAmp})}{\partial t} \right)} > {{AVG\_ SLOPE}{\_ THR}}} \right) \parallel \left( {{\max\left( \frac{\partial({ImpAmp})}{\partial t} \right)} > {SLOPE\_ THR}} \right)} \right)$ Decision = Finger;  Calculate estimated reliability; else  Decision =Spoof;  Calculate estimated reliability;

The method may then set the reliability probability (error function)according to the normalized thresholds/factors, measurements, and slopein case of low reflected acoustic energy, which are referred below asthe three features. The thresholds may then be set during calibrationexcept the sensor amplitude (transmit power), which may also becalculated in real time.

In some embodiments, several parameters may be measured during thecalibration process, which may include but not limited to: 1) calculatethe three features for target pattern/s (calibration at the productionline); 2) calculate the three features for each of the enrollment images(or take specific user image); and/or 3) calculate sine amplitude (maxsample−min sample) using early images.

FIG. 6B illustrates another exemplary implementation of determiningwhether a finger is a spoof according to aspects of the presentdisclosure. As shown in FIG. 6B, in block 606, a controller of theultrasonic fingerprint sensor may be configured to detect a change intemperature of the finger. In block 608, the controller may beconfigured to determine whether the finger is a spoof based at least inpart on the change in temperature of the finger.

FIG. 6C illustrates yet another exemplary implementation of determiningwhether a finger is a spoof according to aspects of the presentdisclosure. In the example shown in FIG. 6C, in block 612, a controllerof the ultrasonic fingerprint sensor may be configured to determinevariations of the reflected acoustic energy of the finger over time. Inblock 614, the controller may be configured to determine whether thefinger is a spoof based at least in part on the variations of thereflected acoustic energy of the finger over time. According to aspectsof the present disclosure, the method performed in block 614 may furtherinclude the methods performed in blocks 616 and 618. In block 616, thecontroller may be configured to compare the variations of the reflectedacoustic energy of the finger over time to variations of the reflectedacoustic energy of an authorized user's finger collected duringenrollment. In block 618, the controller may be configured to determinewhether the finger is a spoof based at least in part on a result of thecomparison.

In some embodiments, a live finger may have dynamic reflected acousticenergy characteristics, resulting in changing reflected acoustic energyover time. The reflected acoustic energy of the finger is measuredmultiple times in order to verify a change of the reflected acousticenergy. Spoofs, on the other hand, usually don't exhibit such a trend,thus can be detected.

According to aspects of the present disclosure, the system may beconfigured to perform early signal calibration. Some of the parameterscan be auto calibrated using the signal at an early time, before thefingerprint information arrives to the receiver. This allows calibrationfor deviations of transmission, reception, strength per pixel. Therun-time values are divided by the early-signal amplitude in order tonormalize the metric.

According to aspects of the present disclosure, the system may beconfigured to use subset of the pixels for liveness and calibration. Theabove methods of the detection may use aggregation of the signal levelsand not fingerprint features. This allows using of sub-images that donot contain the full fingerprint image. The partial image analysis hasseveral benefits: a) It requires less time, thus latency is muchsmaller; b) The power consumption is usually associated with thecapturing and analysis time, and fewer pixels being analyzed means lesspower may be consumed; c) The smaller computational load allows runningthe algorithm on the fingerprint controller, thus further reducingpower, latency, security and complexity. d) It allows self-calibrationdue to parameter change over temperature.

According to aspects of the present disclosure, the sensing of thetemperature of the acoustic load (fingerprint or spoof) can be measuredduring the matching time. The human body has relatively larger mass thana spoof, and is also temperature regulated. This results in acharacteristic heat transfer profile with the fingerprint sensor. Forinstance, a cold sensor will heat up to a human body temperature, andthe same will happen if the sensor is very hot. Spoofs will usually havethe same temperature as their ambient environment. Even if they areheated or cooled for a human body temperature, their smaller mass willusually result in slower slope of the temperature.

According to aspects of the present disclosure, during enrollment of afinger, the different thresholds of the algorithm can be refined. Forinstance, the reflected acoustic energy thresholds can be relaxed orhardened based on the specific finger characteristics. Users that havepoor separation from spoofs metric, can be advised not to use thefeature, in order not to be rejected as spoof “too many times”.

According to aspects of the present disclosure, the level of securitycan be dynamically adjusted per the use case. For monetary transaction,the users may be more tolerant to false rejection as spoof, and havingto try again to match.

FIG. 7 illustrates an exemplary block diagram of a device that mayinclude an ultrasonic fingerprint sensor configured to detect livenessof a finger according to aspects of the present disclosure. The devicethat may be configured to detect liveness of a finger may comprise oneor more features of mobile device 700 shown in FIG. 7. In certainembodiments, mobile device 700 may include a wireless transceiver 721that is capable of transmitting and receiving wireless signals 723 viawireless antenna 722 over a wireless communication network. Wirelesstransceiver 721 may be connected to bus 701 by a wireless transceiverbus interface 720. Wireless transceiver bus interface 720 may, in someembodiments be at least partially integrated with wireless transceiver721. Some embodiments may include multiple wireless transceivers 721 andwireless antennas 722 to enable transmitting and/or receiving signalsaccording to a corresponding multiple wireless communication standardssuch as, for example, versions of IEEE Std. 802.11, CDMA, WCDMA, LTE,UMTS, GSM, AMPS, Zigbee and Bluetooth®, etc.

Mobile device 700 may also comprise GPS receiver 755 capable ofreceiving and acquiring GPS signals 759 via GPS antenna 758. GPSreceiver 755 may also process, in whole or in part, acquired GPS signals759 for estimating a location of a mobile device. In some embodiments,processor(s) 711, memory 740, DSP(s) 712 and/or specialized processors(not shown) may also be utilized to process acquired GPS signals, inwhole or in part, and/or calculate an estimated location of mobiledevice 700, in conjunction with GPS receiver 755. Storage of GPS orother signals may be performed in memory 740 or registers (not shown).

Also shown in FIG. 7, mobile device 700 may comprise digital signalprocessor(s) (DSP(s)) 712 connected to the bus 701 by a bus interface710, processor(s) 711 connected to the bus 701 by a bus interface 710and memory 740. Bus interface 710 may be integrated with the DSP(s) 712,processor(s) 711 and memory 740. In various embodiments, functions maybe performed in response execution of one or more machine-readableinstructions stored in memory 740 such as on a computer-readable storagemedium, such as RAM, ROM, FLASH, or disc drive, just to name a fewexamples. The one or more instructions may be executable by processor(s)711, specialized processors, or DSP(s) 712. Memory 740 may comprise anon-transitory processor-readable memory and/or a computer-readablememory that stores software code (programming code, instructions, etc.)that are executable by processor(s) 711 and/or DSP(s) 712 to performfunctions described herein. In a particular implementation, wirelesstransceiver 721 may communicate with processor(s) 711 and/or DSP(s) 712through bus 701 to enable mobile device 700 to be configured as awireless station. Processor(s) 711 and/or DSP(s) 712 may executeinstructions to execute one or more aspects of processes/methodsdiscussed in connection with FIGS. 1A-1C through FIGS. 6A-6C.Processor(s) 711 and/or DSP(s) 712 may perform the methods and functionsas a part of the controller described in FIGS. 1A-1C through FIGS.6A-6C.

Also shown in FIG. 7, a user interface 735 may comprise any one ofseveral devices such as, for example, a speaker, microphone, displaydevice, vibration device, keyboard, touch screen, etc. A user interfacesignal provided to a user may be one or more outputs provided by any ofthe speaker, microphone, display device, vibration device, keyboard,touch screen, etc. In a particular implementation, user interface 735may enable a user to interact with one or more applications hosted onmobile device 700. For example, devices of user interface 735 may storeanalog or digital signals on memory 740 to be further processed byDSP(s) 712 or processor 711 in response to action from a user.Similarly, applications hosted on mobile device 700 may store analog ordigital signals on memory 740 to present an output signal to a user. Inanother implementation, mobile device 700 may optionally include adedicated audio input/output (I/O) device 770 comprising, for example, adedicated speaker, microphone, digital to analog circuitry, analog todigital circuitry, amplifiers and/or gain control. In anotherimplementation, mobile device 700 may comprise touch sensors 762responsive to touching or pressure on a keyboard or touch screen device.

Mobile device 700 may also comprise a dedicated camera device 764 forcapturing still or moving imagery. Dedicated camera device 764 maycomprise, for example an imaging sensor (e.g., charge coupled device orCMOS imager), lens, analog to digital circuitry, frame buffers, etc. Inone implementation, additional processing, conditioning, encoding orcompression of signals representing captured images may be performed atprocessor 711 or DSP(s) 712. Alternatively, a dedicated video processor768 may perform conditioning, encoding, compression or manipulation ofsignals representing captured images. Additionally, dedicated videoprocessor 768 may decode/decompress stored image data for presentationon a display device (not shown) on mobile device 700.

Mobile device 700 may also comprise sensors 760 coupled to bus 701 whichmay include, for example, inertial sensors and environment sensors.Inertial sensors of sensors 760 may comprise, for example accelerometers(e.g., collectively responding to acceleration of mobile device 700 inthree dimensions), one or more gyroscopes or one or more magnetometers(e.g., to support one or more compass applications). Environment sensorsof mobile device 700 may comprise, for example, temperature sensors,barometric pressure sensors, ambient light sensors, and camera imagers,microphones, just to name few examples. Sensors 760 may generate analogor digital signals that may be stored in memory 740 and processed byDPS(s) or processor 711 in support of one or more applications such as,for example, applications directed to positioning or navigationoperations.

In a particular implementation, mobile device 700 may comprise adedicated modem processor 766 capable of performing baseband processingof signals received and down-converted at wireless transceiver 721 orGPS receiver 755. Similarly, dedicated modem processor 766 may performbaseband processing of signals to be up-converted for transmission bywireless transceiver 721. In alternative implementations, instead ofhaving a dedicated modem processor, baseband processing may be performedby a processor or DSP (e.g., processor 711 or DSP(s) 712).

FIGS. 8A-8C illustrate an example of an ultrasonic sensor according toaspects of the present disclosure. As shown in FIG. 8A, ultrasonicsensor 10 may include an ultrasonic transmitter 20 and an ultrasonicreceiver 30 under a platen 40. The ultrasonic transmitter 20 may be apiezoelectric transmitter that can generate ultrasonic waves 21 (seeFIG. 8B). The ultrasonic receiver 30 may include a piezoelectricmaterial and an array of pixel circuits disposed on a substrate. Inoperation, the ultrasonic transmitter 20 generates one or moreultrasonic waves that travel through the ultrasonic receiver 30 to theexposed surface 42 of the platen 40. At the exposed surface 42 of theplaten 40, the ultrasonic energy may be transmitted, absorbed orscattered by an object 25 that is in contact with the platen 40, such asthe skin of a fingerprint ridge 28, or reflected back. In thoselocations where air contacts the exposed surface 42 of the platen 40,e.g., valleys 27 between fingerprint ridges 28, most of the ultrasonicwave will be reflected back toward the ultrasonic receiver 30 fordetection (see FIG. 8C). Control electronics 50 may be coupled to theultrasonic transmitter 20 and ultrasonic receiver 30 and may supplytiming signals that cause the ultrasonic transmitter 20 to generate oneor more ultrasonic waves 21. The control electronics 50 may then receivesignals from the ultrasonic receiver 30 that are indicative of reflectedultrasonic energy 23. The control electronics 50 may use output signalsreceived from the ultrasonic receiver 30 to construct a digital image ofthe object 25. In some implementations, the control electronics 50 mayalso, over time, successively sample the output signals to detect thepresence and/or movement of the object 25.

According to aspects of the present disclosure, an ultrasonic sensor mayinclude an ultrasonic transmitter 20 and an ultrasonic receiver 30 undera platen 40. The ultrasonic transmitter 20 may be a plane wave generatorincluding a substantially planar piezoelectric transmitter layer.Ultrasonic waves may be generated by applying a voltage to thepiezoelectric layer to expand or contract the layer, depending upon thesignal applied, thereby generating a plane wave. The voltage may beapplied to the piezoelectric transmitter layer via a first transmitterelectrode and a second transmitter electrode. In this fashion, anultrasonic wave may be made by changing the thickness of the layer via apiezoelectric effect. This ultrasonic wave travels toward a finger (orother object to be detected), passing through the platen 40. A portionof the wave not absorbed or transmitted by the object to be detected maybe reflected so as to pass back through the platen 40 and be received bythe ultrasonic receiver 30. The first and second transmitter electrodesmay be metallized electrodes, for example, metal layers that coatopposing sides of the piezoelectric transmitter layer.

The ultrasonic receiver 30 may include an array of pixel circuitsdisposed on a substrate, which also may be referred to as a backplane,and a piezoelectric receiver layer. In some implementations, each pixelcircuit may include one or more TFT elements, electrical interconnecttraces and, in some implementations, one or more additional circuitelements such as diodes, capacitors, and the like. Each pixel circuitmay be configured to convert an electric charge generated in thepiezoelectric receiver layer proximate to the pixel circuit into anelectrical signal. Each pixel circuit may include a pixel inputelectrode that electrically couples the piezoelectric receiver layer tothe pixel circuit.

In the illustrated implementation, a receiver bias electrode is disposedon a side of the piezoelectric receiver layer proximal to platen 40. Thereceiver bias electrode may be a metallized electrode and may begrounded or biased to control which signals are passed to the TFT array.Ultrasonic energy that is reflected from the exposed (top) surface 42 ofthe platen 40 is converted into localized electrical charges by thepiezoelectric receiver layer. These localized charges are collected bythe pixel input electrodes and are passed on to the underlying pixelcircuits. The charges may be amplified by the pixel circuits andprovided to the control electronics, which processes the output signals.A simplified schematic of an example pixel circuit is shown in FIG. 9A,however one of ordinary skill in the art will appreciate that manyvariations of and modifications to the example pixel circuit shown inthe simplified schematic may be contemplated.

Control electronics 50 may be electrically connected to the firsttransmitter electrode and the second transmitter electrode, as well asto the receiver bias electrode and the pixel circuits on the substrate.The control electronics 50 may operate substantially as discussedpreviously with respect to FIGS. 8A-8C.

The platen 40 may be any appropriate material that can be acousticallycoupled to the receiver, with examples including plastic, ceramic,glass, sapphire, stainless steel, a metal alloy, polycarbonate, apolymeric material, or a metal-filled plastic. In some implementations,the platen 40 can be a cover plate, e.g., a cover glass or a lens glassfor a display device or an ultrasonic button. Detection and imaging canbe performed through relatively thick platens if desired, e.g., 3 mm andabove.

Examples of piezoelectric materials that may be employed according tovarious implementations include piezoelectric polymers havingappropriate acoustic properties, for example between about 2.5 MRaylsand 5 MRayls. Specific examples of piezoelectric materials that may beemployed include ferroelectric polymers such as polyvinylidene fluoride(PVDF) and polyvinylidene fluoride-trifluoroethylene (PVDF-TrFE)copolymers. Examples of PVDF copolymers include 60:40 (molar percent)PVDF-TrFE, 70:30 PVDF-TrFE, 80:20 PVDF-TrFE, and 90:10 PVDR-TrFE. Otherexamples of piezoelectric materials that may be employed includepolyvinylidene chloride (PVDC) homopolymers and copolymers,polytetrafluoroethylene (PTFE) homopolymers and copolymers, anddiisopropylammonium bromide (DIPAB).

The thickness of each of the piezoelectric transmitter layer and thepiezoelectric receiver layer may be selected so as to be suitable forgenerating and receiving ultrasonic waves. In one example, a PVDFpiezoelectric transmitter layer is approximately 28 μm thick and aPVDF-TrFE receiver layer is approximately 12 μm thick. Examplefrequencies of the ultrasonic waves are in the range of 5 MHz to 30 MHz,with wavelengths on the order of a quarter of a millimeter or less.

FIGS. 8A-8C show example arrangements of ultrasonic transmitters andreceivers in an ultrasonic sensor, with other arrangements possible. Forexample, in some implementations, the ultrasonic transmitter 20 may beabove the ultrasonic receiver 30, i.e., closer to the object ofdetection. In some implementations, the ultrasonic sensor may include anacoustic delay layer. For example, an acoustic delay layer can beincorporated into the ultrasonic sensor 10 between the ultrasonictransmitter 20 and the ultrasonic receiver 30. An acoustic delay layercan be employed to adjust the ultrasonic pulse timing, and at the sametime electrically insulate the ultrasonic receiver 30 from theultrasonic transmitter 20. The delay layer may have a substantiallyuniform thickness, with the material used for the delay layer and/or thethickness of the delay layer selected to provide a desired delay in thetime for reflected ultrasonic energy to reach the ultrasonic receiver30. In doing so, the range of time during which an energy pulse thatcarries information about the object by virtue of having been reflectedby the object may be made to arrive at the ultrasonic receiver 30 duringa time range when it is unlikely that energy reflected from other partsof the ultrasonic sensor 10 is arriving at the ultrasonic receiver 30.In some implementations, the TFT substrate and/or the platen 40 mayserve as an acoustic delay layer.

FIG. 9A depicts a 4×4 pixel array of pixels for an ultrasonic sensor.Each pixel may, for example, be associated with a local region ofpiezoelectric sensor material, a peak detection diode and a readouttransistor; many or all of these elements may be formed on or in thebackplane to form the pixel circuit. In practice, the local region ofpiezoelectric sensor material of each pixel may transduce receivedultrasonic energy into electrical charges. The peak detection diode mayregister the maximum amount of charge detected by the local region ofpiezoelectric sensor material. Each row of the pixel array may then bescanned, e.g., through a row select mechanism, a gate driver, or a shiftregister, and the readout transistor for each column may be triggered toallow the magnitude of the peak charge for each pixel to be read byadditional circuitry, e.g., a multiplexer and an A/D converter. Thepixel circuit may include one or more TFTs to allow gating, addressing,and resetting of the pixel.

Each pixel circuit may provide information about a small portion of theobject detected by the ultrasonic sensor 10. While, for convenience ofillustration, the example shown in FIG. 9A is of a relatively coarseresolution, ultrasonic sensors having a resolution on the order of 500pixels per inch or higher that are configured with a layered structure.The detection area of the ultrasonic sensor 10 may be selected dependingon the intended object of detection. For example, the detection area mayrange from about 5 mm×5 mm for a single finger to about 3 inches×3inches for four fingers. Smaller and larger areas, including square,rectangular and non-rectangular geometries, may be used as appropriatefor the object.

FIG. 9B shows an example of a high-level block diagram of an ultrasonicsensor system. Many of the elements shown may form part of controlelectronics 50. A sensor controller may include a control unit that isconfigured to control various aspects of the sensor system, e.g.,ultrasonic transmitter timing and excitation waveforms, bias voltagesfor the ultrasonic receiver and pixel circuitry, pixel addressing,signal filtering and conversion, readout frame rates, and so forth. Thesensor controller may also include a data processor that receives datafrom the ultrasonic sensor circuit pixel array. The data processor maytranslate the digitized data into image data of a fingerprint or formatthe data for further processing.

For example, the control unit may send a transmitter (Tx) excitationsignal to a Tx driver at regular intervals to cause the Tx driver toexcite the ultrasonic transmitter and produce planar ultrasonic waves.The control unit may send level select input signals through a receiver(Rx) bias driver to bias the receiver bias electrode and allow gating ofacoustic signal detection by the pixel circuitry. A demultiplexer may beused to turn on and off gate drivers that cause a particular row orcolumn of sensor pixel circuits to provide sensor output signals. Outputsignals from the pixels may be sent through a charge amplifier, a filtersuch as an RC filter or an anti-aliasing filter, and a digitizer to thedata processor. Note that portions of the system may be included on theTFT backplane and other portions may be included in an associatedintegrated circuit.

Having described in some detail an example ultrasonic fingerprintsensor, the following discussion addresses characteristics of typicaldisplay modules. There are many different technologies that may be usedto provide modern, pixelated display devices for use in computermonitors, televisions, mobile devices, and other electronic equipment.Liquid crystal displays (LCDs) and organic light-emitting diode (OLED)displays are examples of two such technologies. As mentioned previously,many of the examples in this disclosure focus on integration of anultrasonic fingerprint sensor with an LCD-type display architecture,although the general techniques, design rules, and concepts outlinedherein may also be applied to other types of display technology as well.

In LCDs, light emitted from a uniformly-illuminated backlight passesthrough two polarizers that are parallel to one another but orientedwith their polarization axes perpendicular to one another. An array ofliquid crystal cells, or pixels, is interposed between the twopolarizers. Each liquid crystal cell is typically configured such thatthe liquid crystal inside “relaxes” into a “twisted nematic state” whenno voltage is applied to the liquid crystal cell. In the twisted nematicstate, the liquid crystal causes polarized light passing through thepolarizer interposed between the liquid crystal cell and the backlightto be twisted by 90°, allowing the light to then pass through theremaining polarizer.

When a voltage is applied across a liquid crystal cell, the liquidcrystal untwists, causing the initially polarized light passing throughthe liquid crystal to be twisted to a lesser degree, resulting in lesstransmission of the light through the second polarizer. The amount oftwisting/untwisting of the light is dependent on the voltage applied,allowing the amount of light that passes through the dual-polarizerstack to be modulated. Each such liquid crystal cell may serve as apixel or a subpixel of a display device. If color output is desired, acolor filter array may be placed between the liquid crystal layer andthe viewing surface of the display. The color filter array may filterthe light that is produced by each pixel such that it is substantiallymonochromatic, e.g., red, green, or blue. By combining the output ofmultiple pixels, e.g., a red pixel, a green pixel, and a blue pixel, itmay be possible to tune the blended color produced by each such pixelgrouping. In such cases, the pixel elements may be referred to assubpixels, and each grouping of subpixels that may be tuned to produceblended light of a particular color may be referred to as a pixel.

OLED displays utilize a more direct technique for providing light. InOLED displays, each pixel, or subpixel, is a single light-emittingdiode. Each diode may be individually controlled so as to produce avarying amount of light of a particular color. This bypasses the needfor polarizer films and liquid crystal elements and reduces the amountof light that is “wasted” by a display panel as compared with an LCDdisplay panel.

While LCDs and OLED displays use very different techniques for producinglight, each type of display requires a mechanism for individuallycontrolling each display pixel or subpixel. To provide such control,these displays utilize an array of thin-film transistors (TFTs). TheTFTs for LCDs are commonly fabricated on a clear TFT backplane (alsoreferred to herein as a backplane), e.g., a glass or transparentpolymer, to facilitate light transmission from the backlight through thebackplane and into the liquid crystal cells. The TFTs for OLED displaysmay also be manufactured on a clear backplane, although opaquebackplanes may be used in such types of displays.

Each display pixel of a display module may include one or more TFTs thatare arranged, sometimes in combination with other circuit elements, in acircuit that controls the behavior of that display pixel; suchpixel-level circuits are referred to herein as display pixel circuits.The display pixel circuits are arranged on the backplane in an arraythat is substantially coextensive with the display pixel array. Ratherthan address all of the display pixel circuits controlling the pixels inthe display simultaneously, which would require separate traces for eachand every display pixel circuit, the control electronics for suchdisplay modules typically sequentially “scan” through each row or columnof the display pixel circuits at a very high frequency. To facilitatesuch control, each column may, for example, have a separate “data” lineor trace, and each row may have a separate “scan” line or trace.Alternatively, each row may have a separate data line or trace, and eachcolumn may have a separate scan line or trace. Each display pixelcircuit may typically be connected to one scan trace and one data trace.Typically, power is applied to the scan traces one at a time and whilepower is applied to a particular scan trace, the display pixel circuitsassociated with the powered scan trace may be individually controlled bysignals applied to their respective data traces.

The use of a scanning arrangement allows the number of individual tracesthat can be accommodated for a display to be reduced from potentiallymillions of traces to hundreds or thousands of traces. This, however, isstill an undesirably large number of traces to deal with, and so displaypanels often include one or more driver chips that communicate with eachdata trace and scan trace and that translate image data provided from aninput or set of inputs into sequential sets of scan signals and datasignals that are output to the scan traces and the data traces. Driverchips are typically connected to a processor or other device thatprovides image data via a flex cable having tens or hundreds ofconductors. Thus, a multimillion pixel display may be controlled by aflexible cable having a drastically lower number of conductors, e.g., onthe order of 4-6 orders of magnitude lower.

Such driver chips may be considerably smaller in footprint than thedisplay may be. To accommodate such a size differential, the spacingbetween the data traces and/or scan traces may be reduced between thedisplay pixel circuit array and the driver chip. From the perspective ofthe driver chip, the traces may appear to “fan out” towards the array ofdisplay pixel circuits, referred to herein as “fanout.” To accommodatethe driver chip or chips and the respective fan-out, the TFT backplanemay be sized larger than the array of display pixel circuits. In somecases, the fanout does not terminate at a driver chip, but insteadterminates at a flex cable connection. The driver chip in such cases maybe located on a component at the opposing terminal end of the flexcable.

Note that the TFT backplane for a display module may, within minimal orno alteration of existing circuit patterning, be designed to accommodatea second array of pixel circuits in the vicinity of the fanout. Such asecond array of pixel circuits may be used to provide ultrasonic sensingfunctionality to a non-display region of the display device;accordingly, the pixel circuits in the second array may be referred toherein as sensor pixel circuits (as opposed to the display pixelcircuits discussed earlier). Such sensing functionality may, forexample, be used to provide an ultrasonic fingerprint sensingcapability. Note that this may be of particular interest in mobileelectronic devices to allow for biometric identification measures to beimplemented in an aesthetically-pleasing manner on the device to helpsecure the device and the data therein in the event of loss or theft.

According to aspects of the present disclosure, ultrasonic sensors canbe configured to produce high-resolution fingerprint images for userverification and authentication. In some implementations, ultrasonicfingerprint sensors can be configured to detect reflected signalsproportional to the differential reflected acoustic energy between anouter surface of a platen and a finger ridge (tissue) and valley (air).For example, a portion of the ultrasonic wave energy of an ultrasonicwave may be transmitted from the sensor into finger tissue in the ridgeareas while the remaining portion of the ultrasonic wave energy isreflected back towards the sensor, whereas a smaller portion of the wavemay be transmitted into the air in the valley regions of the fingerwhile the remaining portion of the ultrasonic wave energy is reflectedback to the sensor. Methods of correcting diffraction effects disclosedherein may increase the overall signal and image contrast from thesensor.

According to aspects of the present disclosure, ultrasonic buttons withfingerprint sensors can be applied for user authentication in a widerange of applications, including mobile phones, tablet computers,wearable devices and medical devices. Ultrasonic authenticating buttonsmay be utilized in personal medical devices such as drug deliverydevices. These devices may be wirelessly connected to track and verifythe identification of a user, type of drug, dosage, time of delivery,and style of delivery. The on-device authenticating button can beconfigured to allow single-user enrollment (e.g., at home or at apharmacy) and local verification for subsequent consumption of the drug.Rapid identification and verification may appear seamless with thedelivery of the drug, as depressions of the ultrasonic button can beconfigured to invoke user verification and drug delivery.Mobile-connected authenticated drug delivery devices may includepersonalized pen-injectors and inhalers. Connected injector pens,inhalers and other medical devices may incorporate an ultrasonic buttonfor patient identification and verification.

Note that at least the following three paragraphs, FIGS. 1A-1B throughFIGS. 9A-9B and their corresponding descriptions provide transmittermeans for transmitting an ultrasonic wave to a finger; receiver meansfor receiving a reflected ultrasonic wave from the finger; controllermeans for determining a reflected acoustic energy of the finger based ona difference between average amplitudes of the reflected ultrasonic wavefrom ridges and valleys of the finger, and determining whether thefinger is a spoof based at least in part on the reflected acousticenergy of the finger; means for estimating a background energy receivedby the receiver means without the finger; means for removing thebackground energy from the reflected acoustic energy of the finger;means for detecting diffractions of the reflected ultrasonic wave; meansfor calculating an image with reduced effects of the diffractions of thereflected ultrasonic wave; means for detecting a non-uniform spatialresponse in the reflected ultrasonic wave; means for adjusting thereflected acoustic energy of the finger to equalize effects of thenon-uniform spatial response in the reflected ultrasonic wave; means foridentifying regions representing ridges and valleys of the finger; meansfor determining the reflected acoustic energy of the finger based on thedifference between average amplitudes of the reflected ultrasonic wavefrom the regions representing the ridges and the valleys of the finger;means for recording a number of early cycles of the ultrasonic wavereceived at the receiver means; means for adjusting the reflectedacoustic energy of the finger based on the number of early cycles of theultrasonic wave received at the receiver means; means for comparing thereflected acoustic energy of the finger to a threshold range; means fordetermining whether the finger is a spoof based at least in part onwhether the reflected acoustic energy of the finger falls within thethreshold range; means for determining variations of the reflectedacoustic energy of the finger over time; means for determining whetherthe finger is a spoof based at least in part on the variations of thereflected acoustic energy of the finger over time; means for detecting achange in temperature of the finger; and means for determining whetherthe finger is a spoof based at least in part on the change intemperature of the finger.

The methodologies described herein may be implemented by various meansdepending upon applications according to particular examples. Forexample, such methodologies may be implemented in hardware, andfirmware/software. In a hardware implementation, for example, aprocessing unit may be implemented within one or more applicationspecific integrated circuits (“ASICs”), digital signal processors(“DSPs”), digital signal processing devices (“DSPDs”), programmablelogic devices (“PLDs”), field programmable gate arrays (“FPGAs”),processors, controllers, micro-controllers, microprocessors, electronicdevices, other devices units designed to perform the functions describedherein, or combinations thereof.

Some portions of the detailed description included herein are presentedin terms of algorithms or symbolic representations of operations onbinary digital signals stored within a memory of a specific apparatus orspecial purpose computing device or platform. In the context of thisparticular specification, the term specific apparatus or the likeincludes a general purpose computer once it is programmed to performparticular operations pursuant to instructions from program software.Algorithmic descriptions or symbolic representations are examples oftechniques used by those of ordinary skill in the signal processing orrelated arts to convey the substance of their work to others skilled inthe art. An algorithm is here, and generally, is considered to be aself-consistent sequence of operations or similar signal processingleading to a desired result. In this context, operations or processinginvolve physical manipulation of physical quantities. Typically,although not necessarily, such quantities may take the form ofelectrical or magnetic signals capable of being stored, transferred,combined, compared or otherwise manipulated. It has proven convenient attimes, principally for reasons of common usage, to refer to such signalsas bits, data, values, elements, symbols, characters, terms, numbers,numerals, or the like. It should be understood, however, that all ofthese or similar terms are to be associated with appropriate physicalquantities and are merely convenient labels. Unless specifically statedotherwise, as apparent from the discussion herein, it is appreciatedthat throughout this specification discussions utilizing terms such as“processing,” “computing,” “calculating,” “determining” or the likerefer to actions or processes of a specific apparatus, such as a specialpurpose computer, special purpose computing apparatus or a similarspecial purpose electronic computing device. In the context of thisspecification, therefore, a special purpose computer or a similarspecial purpose electronic computing device is capable of manipulatingor transforming signals, typically represented as physical electronic ormagnetic quantities within memories, registers, or other informationstorage devices, transmission devices, or display devices of the specialpurpose computer or similar special purpose electronic computing device.

Wireless communication techniques described herein may be in connectionwith various wireless communications networks such as a wireless widearea network (“WWAN”), a wireless local area network (“WLAN”), awireless personal area network (WPAN), and so on. The term “network” and“system” may be used interchangeably herein. A WWAN may be a CodeDivision Multiple Access (“CDMA”) network, a Time Division MultipleAccess (“TDMA”) network, a Frequency Division Multiple Access (“FDMA”)network, an Orthogonal Frequency Division Multiple Access (“OFDMA”)network, a Single-Carrier Frequency Division Multiple Access (“SC-FDMA”)network, or any combination of the above networks, and so on. A CDMAnetwork may implement one or more radio access technologies (“RATs”)such as cdma2000, Wideband-CDMA (“W-CDMA”), to name just a few radiotechnologies. Here, cdma2000 may include technologies implementedaccording to IS-95, IS-2000, and IS-856 standards. A TDMA network mayimplement Global System for Mobile Communications (“GSM”), DigitalAdvanced Mobile Phone System (“D-AMPS”), or some other RAT. GSM andW-CDMA are described in documents from a consortium named “3rdGeneration Partnership Project” (“3GPP”). Cdma2000 is described indocuments from a consortium named “3rd Generation Partnership Project 2”(“3GPP2”). 3GPP and 3GPP2 documents are publicly available. 4G Long TermEvolution (“LTE”) communications networks may also be implemented inaccordance with claimed subject matter, in an aspect. A WLAN maycomprise an IEEE 802.11x network, and a WPAN may comprise a Bluetooth®network, an IEEE 802.15x, for example. Wireless communicationimplementations described herein may also be used in connection with anycombination of WWAN, WLAN or WPAN.

In another aspect, as previously mentioned, a wireless transmitter oraccess point may comprise a femtocell, utilized to extend cellulartelephone service into a business or home. In such an implementation,one or more mobile devices may communicate with a femtocell via a codedivision multiple access (“CDMA”) cellular communication protocol, forexample, and the femtocell may provide the mobile device access to alarger cellular telecommunication network by way of another broadbandnetwork such as the Internet.

Techniques described herein may be used with a GPS that includes any oneof several GNSS and/or combinations of GNSS. Furthermore, suchtechniques may be used with positioning systems that utilize terrestrialtransmitters acting as “pseudolites”, or a combination of satellitevehicles (SVs) and such terrestrial transmitters. Terrestrialtransmitters may, for example, include ground-based transmitters thatbroadcast a PN code or other ranging code (e.g., similar to a GPS orCDMA cellular signal). Such a transmitter may be assigned a unique PNcode so as to permit identification by a remote receiver. Terrestrialtransmitters may be useful, for example, to augment a GPS in situationswhere GPS signals from an orbiting SV might be unavailable, such as intunnels, mines, buildings, urban canyons or other enclosed areas.Another implementation of pseudolites is known as radio-beacons. Theterm “SV”, as used herein, is intended to include terrestrialtransmitters acting as pseudolites, equivalents of pseudolites, andpossibly others. The terms “GPS signals” and/or “SV signals”, as usedherein, is intended to include GPS-like signals from terrestrialtransmitters, including terrestrial transmitters acting as pseudolitesor equivalents of pseudolites.

The terms, “and,” and “or” as used herein may include a variety ofmeanings that will depend at least in part upon the context in which itis used. Typically, “or” if used to associate a list, such as A, B or C,is intended to mean A, B, and C, here used in the inclusive sense, aswell as A, B or C, here used in the exclusive sense. Referencethroughout this specification to “one example” or “an example” meansthat a particular feature, structure, or characteristic described inconnection with the example is included in at least one example ofclaimed subject matter. Thus, the appearances of the phrase “in oneexample” or “an example” in various places throughout this specificationare not necessarily all referring to the same example. Furthermore, theparticular features, structures, or characteristics may be combined inone or more examples. Examples described herein may include machines,devices, engines, or apparatuses that operate using digital signals.Such signals may comprise electronic signals, optical signals,electromagnetic signals, or any form of energy that provides informationbetween locations.

While there has been illustrated and described what are presentlyconsidered to be example features, it will be understood by thoseskilled in the art that various other modifications may be made, andequivalents may be substituted, without departing from claimed subjectmatter. Additionally, many modifications may be made to adapt aparticular situation to the teachings of claimed subject matter withoutdeparting from the central concept described herein. Therefore, it isintended that claimed subject matter not be limited to the particularexamples disclosed, but that such claimed subject matter may alsoinclude all aspects falling within the scope of the appended claims, andequivalents thereof.

We claim:
 1. A method of detecting a spoof finger by an ultrasonicfingerprint sensor, comprising: transmitting, by an ultrasonictransmitter of the ultrasonic fingerprint sensor, an ultrasonic wave toa finger; receiving, by an ultrasonic sensor array of the ultrasonicfingerprint sensor, a reflected ultrasonic wave from the finger;determining, by a controller of the ultrasonic fingerprint sensor, areflected acoustic energy of the finger based on a difference betweenaverage amplitudes of the reflected ultrasonic wave from ridges andvalleys of the finger, wherein determining the reflected acoustic energyof the finger comprises: recording a number of early cycles of theultrasonic wave received at the ultrasonic sensor array; and adjustingthe reflected acoustic energy of the finger based on the number of earlycycles of the ultrasonic wave received at the ultrasonic sensor array;and determining, by the controller of the ultrasonic fingerprint sensor,whether the finger is a spoof based at least in part on the reflectedacoustic energy of the finger.
 2. The method of claim 1, whereindetermining the reflected acoustic energy of the finger comprises:estimating a background energy received by the ultrasonic sensor arraywithout the finger; and removing the background energy from thereflected acoustic energy of the finger.
 3. The method of claim 1,wherein determining the reflected acoustic energy of the finger furthercomprises: detecting diffractions of the reflected ultrasonic wave; andcalculating an image with reduced effects of the diffractions of thereflected ultrasonic wave.
 4. The method of claim 1, wherein determiningthe reflected acoustic energy of the finger further comprises: detectinga non-uniform spatial response in the reflected ultrasonic wave; andadjusting the reflected acoustic energy of the finger to equalizeeffects of the non-uniform spatial response in the reflected ultrasonicwave.
 5. The method of claim 1, wherein determining the reflectedacoustic energy of the finger further comprises: identifying regionsrepresenting ridges and valleys of the finger; and determining thereflected acoustic energy of the finger based on the difference betweenaverage amplitudes of the reflected ultrasonic wave from the regionsrepresenting the ridges and the valleys of the finger.
 6. The method ofclaim 1, wherein determining whether the finger is a spoof comprises:comparing the reflected acoustic energy of the finger to a thresholdrange; and determining whether the finger is a spoof based at least inpart on whether the reflected acoustic energy of the finger falls withinthe threshold range.
 7. The method of claim 1, wherein determiningwhether the finger is a spoof further comprises: determining variationsof the reflected acoustic energy of the finger over time; anddetermining whether the finger is a spoof based at least in part on thevariations of the reflected acoustic energy of the finger over time. 8.The method of claim 7, further comprising: comparing the variations ofthe reflected acoustic energy of the finger over time to variations ofthe reflected acoustic energy of an authorized user's finger collectedduring enrollment; and determining whether the finger is a spoof basedat least in part on a result of the comparison.
 9. The method of claim1, wherein determining whether the finger is a spoof further comprises:detecting a change in temperature of the finger; and determining whetherthe finger is a spoof based at least in part on the change intemperature of the finger.
 10. An ultrasonic fingerprint sensor,comprising: an ultrasonic transmitter configured to transmit anultrasonic wave to a finger; an ultrasonic sensor array configured toreceive a reflected ultrasonic wave from the finger; and a controllerconfigured to: determine a reflected acoustic energy of the finger basedon a difference between average amplitudes of the reflected ultrasonicwave from ridges and valleys of the finger, record a number of earlycycles of the ultrasonic wave received at the ultrasonic sensor array,adjust the reflected acoustic energy of the finger based on the numberof early cycles of the ultrasonic wave received at the ultrasonic sensorarray, and determine whether the finger is a spoof based at least inpart on the reflected acoustic energy of the finger.
 11. The ultrasonicfingerprint sensor of claim 10, wherein the controller is furtherconfigured to: estimate a background energy received by the ultrasonicsensor array without the finger; and remove the background energy fromthe reflected acoustic energy of the finger.
 12. The ultrasonicfingerprint sensor of claim 10, wherein the controller is furtherconfigured to: detect diffractions of the reflected ultrasonic wave; andcalculate an image with reduced effects of the diffractions of thereflected ultrasonic wave.
 13. The ultrasonic fingerprint sensor ofclaim 10, wherein the controller is further configured to: detect anon-uniform spatial response in the reflected ultrasonic wave; andadjust the reflected acoustic energy of the finger to equalize effectsof the non-uniform spatial response in the reflected ultrasonic wave.14. The ultrasonic fingerprint sensor of claim 10, wherein thecontroller is further configured to: identify regions representingridges and valleys of the finger; and determine the reflected acousticenergy of the finger based on the difference between average amplitudesof the reflected ultrasonic wave from the regions representing theridges and the valleys of the finger.
 15. The ultrasonic fingerprintsensor of claim 10, wherein the controller is further configured to:compare the reflected acoustic energy of the finger to a thresholdrange; and determine whether the finger is a spoof based at least inpart on whether the reflected acoustic energy of the finger falls withinthe threshold range.
 16. The ultrasonic fingerprint sensor of claim 10,wherein the controller is further configured to: determine variations ofthe reflected acoustic energy of the finger over time; and determinewhether the finger is a spoof based at least in part on the variationsof the reflected acoustic energy of the finger over time.
 17. Theultrasonic fingerprint sensor of claim 16, wherein the controller isfurther configured to: compare the variations of the reflected acousticenergy of the finger over time to variations of the reflected acousticenergy of an authorized user's finger collected during enrollment; anddetermine whether the finger is a spoof based at least in part on aresult of the comparison.
 18. The ultrasonic fingerprint sensor of claim10, wherein the controller is further configured to: detect a change intemperature of the finger; and determine whether the finger is a spoofbased at least in part on the change in temperature of the finger. 19.An ultrasonic fingerprint sensor, comprising: transmitter means fortransmitting an ultrasonic wave to a finger; receiver means forreceiving a reflected ultrasonic wave from the finger; and controllermeans for: determining a reflected acoustic energy of the finger basedon a difference between average amplitudes of the reflected ultrasonicwave from ridges and valleys of the finger, wherein the controller meansfor determining the reflected acoustic energy comprises means forrecording a number of early cycles of the ultrasonic wave received atthe receiver means, and means for adjusting the reflected acousticenergy of the finger based on the number of early cycles of theultrasonic wave received at the receiver means; and controller means fordetermining whether the finger is a spoof based at least in part on thereflected acoustic energy of the finger.
 20. The ultrasonic fingerprintsensor of claim 19, wherein the means for determining the reflectedacoustic energy of the finger comprises: means for estimating abackground energy received by the receiver means without the finger; andmeans for removing the background energy from the reflected acousticenergy of the finger.
 21. The ultrasonic fingerprint sensor of claim 19,wherein the means for determining the reflected acoustic energy of thefinger further comprises: means for detecting diffractions of thereflected ultrasonic wave; and means for calculating an image withreduced effects of the diffractions of the reflected ultrasonic wave.22. The ultrasonic fingerprint sensor of claim 19, wherein the means fordetermining the reflected acoustic energy of the finger furthercomprises: means for detecting a non-uniform spatial response in thereflected ultrasonic wave; and means for adjusting the reflectedacoustic energy of the finger to equalize effects of the non-uniformspatial response in the reflected ultrasonic wave.
 23. The ultrasonicfingerprint sensor of claim 19, wherein the means for determining thereflected acoustic energy of the finger further comprises: means foridentifying regions representing ridges and valleys of the finger; andmeans for determining the reflected acoustic energy of the finger basedon the difference between average amplitudes of the reflected ultrasonicwave from the regions representing the ridges and the valleys of thefinger.
 24. The ultrasonic fingerprint sensor of claim 19, wherein themeans for determining whether the finger is a spoof comprises: means forcomparing the reflected acoustic energy of the finger to a thresholdrange; and means for determining whether the finger is a spoof based atleast in part on whether the reflected acoustic energy of the fingerfalls within the threshold range.
 25. The ultrasonic fingerprint sensorof claim 19, wherein the means for determining whether the finger is aspoof further comprises: means for determining variations of thereflected acoustic energy of the finger over time; and means fordetermining whether the finger is a spoof based at least in part on thevariations of the reflected acoustic energy of the finger over time. 26.The ultrasonic fingerprint sensor of claim 19, wherein the means fordetermining whether the finger is a spoof further comprises: means fordetecting a change in temperature of the finger; and means fordetermining whether the finger is a spoof based at least in part on thechange in temperature of the finger.
 27. A non-transitory medium storinginstructions for execution by one or more processors, the instructionscomprising: instructions for transmitting, by an ultrasonic transmitterof the ultrasonic fingerprint sensor, an ultrasonic wave to a finger;instructions for receiving, by an ultrasonic sensor array of theultrasonic fingerprint sensor, a reflected ultrasonic wave from thefinger; instructions for determining, by a controller of the ultrasonicfingerprint sensor, a reflected acoustic energy of the finger based on adifference between average amplitudes of the reflected ultrasonic wavefrom ridges and valleys of the finger; wherein the instructions fordetermining the reflected acoustic energy comprises instructions forrecording a number of early cycles of the ultrasonic wave received atthe receiver means and instructions for adjusting the reflected acousticenergy of the finger based on the number of early cycles of theultrasonic wave received at the receiver means; and instructions fordetermining, by the controller of the ultrasonic fingerprint sensor,whether the finger is a spoof based at least in part on the reflectedacoustic energy of the finger.